Performance Evaluation of Anomaly-Based Detection Mechanisms

Common practice in anomaly-based intrusion detection is that one size fits all: a single anomaly detector should detect all anomalies. Compensation for any performance shortcomings is sometimes effected by resorting to correlation techniques, which could be seen as making use of detector diversity. Such diversity is intuitively based on the assumption that detector coverage is different – perhaps widely different – for different detectors, each covering some disparate portion of the anomaly space. Diversity, then, enhances detection coverage by combining the coverages of individual detectors across multiple sub-regions of the anomaly space, resulting in an overall detection coverage that is superior to the coverage of any one detector. No studies have been done, however, in which measured effects of diversity in anomaly detectors have been obtained. This paper explores the effects of using diverse anomalydetection algorithms (algorithmic diversity) in intrusion detection. Experimental results indicate that while performance/coverage improvements can in fact be effected by combining diverse detection algorithms, the gains are surprisingly not the result of combining large, non-overlapping regions of the anomaly space. Rather, the gains are seen at the edges of the space, and are heavily dependent on the parameter values of the detectors, as well as on the characteristics of the anomalies. As a consequence of this study, defenders can be provided with detailed knowledge of diverse detectors, how to combine and parameterize them, and under what conditions, to effect diverse detection performance that is superior to the performance of a single detector.